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ABSTRACT 
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California. A survey instrument with multiple choice questions covering five 
liberal arts subject areas was distributed to 4,200 students in randomly 
selected classes at ten colleges. More than 2,500 questionnaires were 
returned, for a 60% response rate. Key findings include: (1) an association 

between age and performance in the content areas was observed--older students 
tended to score higher in English and lower on the mathematics section, which 
may indicate that living longer and having more exposure to cultural and 
written materials enhances scores in English; (2) there was a positive 
relationship between the number of units students’ had completed in a subject 
area and the scores obtained on the corresponding section of the test, a 
relationship that was especially strong in science classes but less 
significant for humanities classes; (3) students’ self-assessments of their 
abilities in liberal arts subjects were remarkably accurate, showing that 
community college students are realistic about their academic skills. 
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Introduction 



Courses in English, humanities, math, science, and social science account for more than 
half the community college credit enrollment, confirming that the liberal arts continue to 
represent a major curricular function of the community college (Striplin, 2000). The benefits of 

I t 

a liberal arts education are pervasive, including the development of critical analysis, reading, and 
writing skills and an appreciation for and understanding of cultures and society (Belcher, 1999). 
A liberal arts education helps develop a free citizenry, capable of weighing complex issues and 
participating in informed community decision making. In addition, liberal arts constitute the 
foundation for transfer to four-year institutions. In sum, the learning and skills acquired through 
a liberal arts education in community colleges foster informed citizens, competent workers, and 
successful students. 

Contemporary calls for institutional accountability typically focus on college-wide 
policies and practices related to measures such as rate of student retention, completion of 
programs, and transfer to four-year colleges and universities. These types of indicators have been 
mandated by state boards in nearly half the states, and in several, special funds have been made 
available to community colleges where students show gains in these areas (Cohen, 2002). 

Increasingly, accrediting agencies have also called for colleges to provide data on student 
learning outcomes (Seybert, 2002; Summers, 1997). Measuring complex learning outcomes is 
key to maximizing the quality of education for diverse community college populations, yet 
defining and then selecting appropriate methods to measure learning has been a challenge for 
two-year colleges. The General Academic Learning Experience (GALE) study demonstrates a 
procedure by which cohort-level student knowledge in the liberal arts can be assessed as a 
function of student characteristics such as the number of courses completed, reasons for 
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attending college, and self-rated abilities of students. Such assessment at the institutional level 

can help to inform decision-making from the classroom to the boardroom. 

: ' ! 

Background 

There are a range of methods that have been used to assess community college student 
learning in the liberal arts, including standardized and locally developed tests, performance- 
based institutional portfolios, and capstone courses (Seybert, 2002). GALE replicates an earlier 
effort, the 1983-84 General Academic Assessment (GAA) study completed by the Center for 
Study of Community Colleges (Cohen and Drawer, 1987). The GAA instrument consisted of a 
demographic survey and 299 multiple choice test questions distributed among five test forms and 
administered to over 8,000 students from 23 colleges. The multiple choice items represented a 
range of knowledge in the liberal arts subject areas of English, humanities, mathematics, sciences 
and social sciences that might be expected of a student who had taken one introductory course in 
the related subject. 

1983-84 GAA findings 

Student knowledge in the liberal arts was found to be positively correlated to age, 
especially in English usage, social sciences and the humanities - the older the student, the higher 
the scores in these subject areas. These age-related differences were not found in mathematics or 
science. All five subject area scores showed a direct positive relationship between the number of 
units students had completed in the subject area and the scores obtained on the corresponding 
sections of the test. For example, students who reported taking more social science courses 
scored higher in the social science sections than students who had taken fewer social science 
courses. This relationship was most evident in the humanities and the least evident in English. 
Finally, GAA findings demonstrated the accuracy of students’ self-assessments. Students were 
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asked to compare themselves with other students at their college in terms of their ability to “use 
algebra to solve problems,” “edit written material,” and demonstrate similar competencies that 
suggested knowledge in the liberal arts. High correlations between students’ self-ratings and test 

scores indicate that community college students were realistic about their academic skills, a 

( 

finding that may prove useful in development of other viable assessment options. 

GALE Methods 
Procedure 

To facilitate development and implementation of the test, GALE participation was 
restricted to community colleges located in Southern California. In January of 2002, over 50 
comrnunity colleges in the Southern California areas were contacted by telephone to assess 
interest in participating in GALE. Over 20 colleges indicated strong or moderate interest in 
participating and 16 sent representatives to a GALE orientation meeting held at UCLA in 
February of 2002. Ten of these colleges chose to participate in the GALE study. 

Development of the GALE instrument 

The current GALE instrument is an updated version of the 1983-84 General Academic 
Assessment (GAA) instrument that was developed for a similar study of student learning. The 
GALE instrument consists of two parts: a 7-question demographic survey and a 50-question 
multiple choice content test split into five liberal arts subject areas (see Table 1 for a listing of 
subdisciplines by liberal arts subject areas). Demographic information requested included age, 
high school grade point average (GPA), total number of college courses completed, highest 
degree earned, reason for attending community college, number of liberal arts courses completed 
(by subject area), and self-assessment of cognitive skills related to the five subject areas (see 
Appendix A for a copy of the demographic questions). 
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Table 1: Listing of subdisciplines with each liberal arts subject area 



Subject area 


Subdisciplines 


English 


Reading, writing, decoding and interpreting/' 
. communication through language 


Humanities 


Foreign languages, history, literature, art 
history/appreciation, music history/appreciation, cultural 
anthropology, religious studies, cultural geography and 
philosophy 


Mathematics 


Introductory and intermediate math, applied (technology 
, related), statistics and probability, computer science and 
technology, remedial math 


Science 


Chemistry, biology, physics, earth and space sciences, 
engineering, environmental sciences, agriculture and 
natural resources, physical anthropology 


Social science 


Economics, archaeology, anthropology, urban planning, 
psychology, sociology, political science 



Multiple choice items 

A bank of 250 liberal arts multiple-choice items was developed by GALE researchers 
with the assistance of faculty from participating colleges. We started with a review and revision 
of the original items from the 1983-84 GAA study. These items had been drawn from several 
sources, including the National Assessment of Educational Progress, Educational Testing 
Service, City Colleges of Chicago and Miami-Dade Community College. GALE researchers 
also drew new items from community college faculty suggestions, placement exams, and 
introductory textbooks in the subject areas. 

GALE items were designed to test the level of knowledge expected of a student who had 
completed one introductory college course in the related subject area. In addition to the sources 
of items and faculty review process described above, we pilot tested two sections of GALE 
(English and humanities) in five summer session classes at a non-participating Southern 
California college. We chose to test English and humanities because items in these two subject 
areas had undergone the most revision compared to the 1984 instrument. The number of 
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responses we received did not allow us to perform an item analysis, but we found students 
missed more humanities than English questions. We subsequently revised those humanities 
questions that most or all students missed, seeking to improve question clarity and distinction 
among answer choices. 

Ultimately, the 250 items developed for GALE were subdivided into five separate test 
forms (A through E). Each test form contained 10 items from each of the five liberal arts content 
areas for a total of 50 items per form. Each of the five 50-item forms had unique questions and 
was designed to be completed in a 50-minute class period. The ordering of the five subject areas 
on the different forms was varied to mitigate the possibility that students would consistently 
underscore on a particular subject area if they ran out of time (see Appendix B for a sample of 
questions by subject area). 

Reliability of the GALE instrument 

A multiple-choice test is considered reliable to the extent it is free from random error 
(Stangor, 1998). To assess the reliability of the GALE instrument, Cronbach’s alphas or 
intercoiTelations for scores on items in each ten-item subject area on each test form were 
calculated (Table 2). All GALE subject areas show acceptable levels of reliability with alphas in 
excess of 0.7. ’ 



' While values as low as 0.5 are considered satisfactory for short tests with 10 to 15 items, values of 0.8 or higher 
are expected of tests with over 50 items (Kehoe, 1995). 
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Table 2: Alpha coefficients for GALE test items 



Alpha Coefficients 



Form 


English 


Humanities ■ 


Math 


Science;’ 


Social Science 


A 


,0.73 


0.81 


0.77' 


0.85 


0.84 


B 


0.84 


0.78 


0.75 


0.75 


0.82 


c* 


0.85 


0.60 


0.89 


0.75 


0.71 


D 


0.79 


0.88 1 


0.87 


0.74 


0.65 ' 


E 


0.77 


0.89 


0.83 


0.88 


0.70 


Average 


0.79 


0.79 


0.82 


0.79 


0.74 


Standard Deviation 


0.05 


0.12 


, 0.06 


0.07 


0.08 



*Form C had five rather than the usual ten humanities items - see Appendix C for a discussion. 



Construct validity 

While reliability is a measure of the extent to which students’ scores are consistent within 
subject areas and free from random error, construct validity refers to the extent that GALE 
measurements (e.g. scores by subject area) actually measure the intended underlying construct 
(e.g. student knowledge in subject areas). It appears that GALE test items have face validity 
since many of the items on this instrument were drawn from an existing instrument further 
informed by community college faculty suggestions, community college placement exam 
questions, and items selected from introductory textbooks in the subject areas. 



Another measure of construct validity is concurrent validity, which checks the correlation 
of a self-reported measure with a behavioral variable that is measured at the same time. As the 
results section will show, students’ self-rated ability in GALE subject area related skills 
correlated in a positive and statistically significant way with subsequent subject area test scores. 
This is another indication that the GALE instrument measures what it intends to measure. 



O 

ERIC 



8 



7 



Power of discrimination 

An instrument used for assessment should further demonstrate the ability to discriminate 
among a sample based on the outcome variable being examined. A number of students scored 
100% on GALE items within a given subject area indicating that all questions were answerable 
and it was possible for students to correctly answer all 50 questions. The mean score for 
performance in each of the subject areas was approximately 50%, a result similar to that reported 
in the 1983-84 study and falling within the range (between 30% and 80% correct) suggested for 
items that show adequate power to distinguish among a sample (Kehoe, 1995). The GALE 
instrument also appears capable of distinguishing between students who have taken classes in a 
specific subject area and those who have not (see Appendix C). 

Random selection of classes and cohort analysis 

The class section was used as the unit of sampling for GALE because it was the most 
feasible way of administering a survey to community college students enrolled in credit classes. 
One college-level class was randomly selected from each page of a college’s fall 2002 class 
schedule, excluding lab classes, distance education classes, internships, non-scheduled classes, 
short courses or weekend classes, physical education classes, classes with field trips, and other 
courses not generally taught in a classroom setting. These types of classes were excluded 
because students might have been physically difficult to locate for the administration of the 
instrument, or in a non-lecture setting where exam taking may be difficult. In addition, ESL 
classes were also excluded because student comprehension of questions could not be assumed. 



^ The 1983-84 GAA was based on items supplied by the National Assessment of Educational Progress, Educational 
Testing Service, City Colleges of Chicago, and Miami-Dade Community College, all considered reliable sources for 
community college test items. 




9 



8 



This sampling method has the disadvantage of skewing the sample toward full-time 
students because a student taking three classes has three times as many chances of being in a 

■ , > ■ ■ I 

sampled class as a student who is taking only one class. Nevertheless, the sample, based on 
duplicated head count, should be an accurate representation of the general student body, since 
full-time students represent higher proportions of the full-time-equivalent enrollment in the 
college. 

The GALE survey combines item sampling with population sampling. Not all students in 
the population take the test; no student responds to all the items. The value of this sampling and 
testing procedure is that testing time is reduced; it becomes possible to administer 250 items as 
well as a brief demographic section to the sample within one class hour. 

A minimum of 12 and 15 classes or 420 students were randomly selected from each of 
the 10 participating Southern California colleges. The actual number of students who 
participated in GALE is summarized in Table 3 by institution. 



Table 3: Distribution of sample by institution 



Institution 


Number of 
participants 


Valid Percent 


1 


303 


12.1 


2 


338 


13.5 


3 


154 


6.1 


4 


266 


10.6 


5 


231 


9.2 


6 


257 


10.3 


7 


296 


11.8 


8 


278 


11.1 


9 


277 


11.0 


10 


107 


4.3 


Total 


2507 


100.0 



The study averaged a 60 percent response rate with 2507 students responding from an instrument 
distribution of 4,200. Given the random selection processes used, we expect even the smallest 
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sample (n=107 from Institution 10) to generalize to the student body with a 90 percent 
confidence level and 10 percent error. The other samples should generalize at least as well for 
the purposes of analysis. 

The median age of the particpants was 21 years (mode=19 years). An overwhelming 
majority (84.7%) of the students sampled had not yet earned any type of degree, and over 65% of 
them indicated that their primary reason for attending community college was to prepare for. 
transfer to a four- year college or university. The median high school grade point average 
reported by these students was a B average (2.75-3.24) 

Aggregate Data Analysis 

The following results reflect an aggregate or cohort analysis of results from all 
participating students. No college, no class, no faculty member and no student are separately 
identified. Instead, this analysis is intended to examine the broad relationships between student 
characteristics (including course-taking histories) and knowledge of the liberal arts. 

Preliminary descriptive analysis of the aggregate data included running frequencies of 
each of the demographic variables to examine characteristics of the students sampled. Students’ 
performance within a given subject area and on the total content items was of primary 
importance in our inquiry of what community college students know. Thus, cross-tabulations 
and correlations were run between student background variables and scores on individual subject 
- areas and on the total aggregate-of-eontent items to see if meaningful relationships existed among 
the variables assessed. 
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Results and Discussion 



Age and GPA I 

An association between age and performance on the content areas was observed (Table 
4). Older students tended to score higher in English and lower on the mathematics sections, 

i ' ’ 

reflecting a similar finding from the 1983-84 GAA. It may be that achievement in math is more 
school-related while scores in English are also enhanced by just having lived longer and having 
been exposed to more cultural and written materials 



Table 4: Correlation of age and high school grade point average (GPA) with performance 
on content items 





N 


English Humanities 


Math 


Science 


Soc. Science 


Total 


Age 


2395 


.070** 


.012 


-111** 


-.026 


.026 


-.014 


High school GPA 


2417 


.121** 


.061** 


.213** 


.116** 


.057** 


.166** 



**Correlation is significant at the 0.01 level 



While many of the figures listed in Table 4 are statistically significant, they are also small. For 
example, community college students’ high school GPA correlated to performance on all 
sections of the content items, with all correlations being positive and statistically significant. 
Conceivably, those students who performed well in high school are also those who have taken 
more academic classes and have higher GPAs while in college and therefore show higher 
achievement in the five subject areas. On the other hand, since these correlations are small, the 
practical influence of other potential predictor variables should be examined. 



Number of courses completed 

The number of courses that students had taken varied by subject area. For example, as 
Table 5 indicates, students reported taking the most courses in social science (including 
economics, archaeology, anthropology, urban planning, psychology, sociology and political 
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science). Students reported taking fewest courses in science (including chemistry, biology, 
physics, earth and space sciences, engineering, environmental sciences, agriculture and natural 
resources and physical anthropology). Table 5 describes overall course taking patterns reported 
by the GALE participants. 



Table 5: Number of courses previously taken according to subject area 



Courses Completed in: 


N 


Mean 


Median 


Standard Deviation 


English 


2275 


1.83 


1.00 


2.13 


Humanities 


2233 


2.00 


1.00 


2.50 


Math 


2228 


2.14 


1.00 


2.55 


Science 


2253 


1.54 


1.00 


2.31 


Social Science 


2267 


2.42 


2.00 


2.63 


Total* 




14.78 


9.00 


17.87 



*Note “total” refers to the total number of courses that students reported taking at this or any 
college, not the sum of the mean, median or standard deviation. 



Thus, on average we might expect students in this sample to score higher on social science and 
math than science or English classes. However, reference to the median and standard deviation 
figures suggest that the data are quite dispersed and that measures other than the mean number of 
courses taken will be needed to characterize student knowledge. 

Uniformly across subject areas. Table 6 results show that the scores from students who 
have taken three or more courses in a subject area were higher than those who had taken none. 



Table 6: Mean scores by number of total completed courses in scale-related curriculum 



GALE Scale* 


Courses 
Completed in: 


English 


Humanities 


Mathematics 


Science 


Social 

Science 


None 


5.28 


3.51 


4.56 


3.75 


4.74 


One 


6.01 


3.89 


5.06 


4.41 


5.21 


Two 


5.97 


3.60 


5.43 


4.43 


5.24 


Three or more 


5.82 


4.16 


5.93 


4.94 


5.50 



*The GALE Scale represents the mean number of items on a ten point scale answered correctly. 
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For example, students who had taken more classes in science scored higher on the science 
subject area than their peers who had taken no science classes. Similarly, students who had 
completed a greater total number of college courses regardless of subject areas typically scored 
higher overall and higher in most subject areas compared to students who had taken no courses 
(Table 7) - a finding reflected by the 1983-84 GAA study. 

GALE humanities subject area results represent an aberration from this pattern (Table 7). 
Even students who reported taking more than ten college humanities courses scored little higher 
on the humanities subject test than student who had never taken a single course (3.80 versus 3.88 
mean number of correct answers). Unlike the four other subject areas, the total liberal arts test 
scores, and the findings from the 1983-84 study, humanities test scores correlate poorly with the 
total number of non-specific courses that a student had already completed (Table 7). 



Table 7: Mean scores by total number of completed courses 



Completed 

Courses 

(total) 






GALE Scale* 








English 


Humanities 


Mathematics 


Science 


Social 

Science 


Total 

Liberal Arts 


0 


5.18 


3.80 


4.79 


3.93 


4.74 


22.01 


1-3 


5.61 


3.70 


4.81 


4.33 


5.06 


23.08 


4-6 


5.66 


3.55 


4.75 


3.89 


5.21 


22.74 


7-9 


5.95 


3.68 


5.13 


4.21 


5.35 


23.97 


10 or more 


5.94 


3.88 


5.60 


4.26 


5.27 


24.61 



*The GALE Scale represents the mean number of items on a ten-point scale answered correctly. 



Most important reason for attending college 

Students were asked to choose one of the following to characterize their most important 
reason for attending college at this time: (1) to prepare for transfer to a four-year college or 
university; (2) to gain skills necessary to enter a specific occupation; (3) to gain skills necessary 
to advance in a current occupation; (4) to improve my English, reading or math skills; or (5) to 
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satisfy a personal interest. As indicated in Table 8, mean test scores vary according to the most 
important reason students indicated. 



Table 8: Mean test scores by reason for attending college 











GALE Scale* 








Most important reason 












Social 




for attending college 


N 


English 


Humanities Math 


Science 


Science 


Total 


Prepare for transfer 


1531 


5.72 


3.82 


5.50 


4.22 


5.20 


4.9 


Gain specific skills to 
enter a specific 
occupation 


373 


5.75 


3.56 


4.52 


4.03 


5.08 


4.60 


Gain skills necessary to 
advance in current 
occupation 


145 


5.37 


3.46 


4.27 


3.81 


4.86 


4.37 


To improve my 
English, reading or 
math skills 


46 


2.94 


2.46 


3.13 


3.00 


3.33 


2.98 


Personal interest 


136 


5.91 


4.05 


4.73 


4.24 


5.09 


4.81 


Aggregate mean score 


2507 


5.62 


3.71 


5.15 


4.11 


5.12 


4.76 


(standard deviation) 




(2.80) 


(2.75) 


(3.38) 


(2.80) 


(3.03) 


(2.10) 



*The GALE Scale represents the mean number of items on a ten-point scale answered correctly. 



The 1983-84 GAA results indicated that students attending college for their personal 
interest scored highest on every subject area, with those preparing for transfer making the 
second-highest scores overall (Cohen and Drawer, 1987). As the last column of Table 8 
indicates, students who were preparing for transfer scored highest overall on average on GALE, 
followed closely by students who are attending college to pursue a personal interest. Students 
preparing for transfer scored highest in the individual subject areas of math and social science, 
while those who are attending college to satisfy a personal interest scored highest on English, 
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humanities and science. Students seeking to gain specific skills to enter a specific occupation 

scored higher in each subject area than students who were seeking to gain skills necessary to 

' / 

advance in a current occupation. Finally, as might be expected, students seeking to improve their 

English, reading or math skills tended to score the lowest on all subject areas in GALE. 

' ' ' . ' 

Self-assessment of abilities 

Before taking the GALE subject area tests, students were asked to rate their abilities 
relative to other students at their college as “poor,” “fair,” “good,” or “excellent” in the following 
six areas: (1) understanding the implications of scientific and technological developments; (2) 
understanding art, classical music, drama; (3) understanding different political ideologies; (4) 
editing written material; (5) solving mathematical equations; and (6) using logical reasoning to 
solve problems (see Appendix A). As depicted in Table 9 below, students were accurate in their 
self-assessments - a finding that is in keeping with the 1983-84 GAA study. 



Table 9: Correlation of self-rated ability with performance on content items 



Self-Rated Ability to: 


N 


English 


Humanities 


Math 


'Science 


Soc. Science 


Total 


Edit written material 


2399 


226** 


.157** 


.046* 




.120** 


.183** 


Understand art, classical 
music, drama 


2401 


.118** 


j23** 


-.056** 


.061** 


.077** 


.082** 


Solve mathematical 
equations 


2392 


.058** 


.050* 


.332** 


.135** 


.045* 


j 9 j** 


Understand implications o 
scientific technology 
development 


2408 


.200** 


.164** 


.223** 


.233** 


.154** 


.278** 


Understand different 
political ideologies 


2389 


.190** 


.200** 


091** 


.176** 


.167** 


229** 


Use logical reasoning to 
solve problems 


2405 


.176** 


.145** 


.136** 


J74** 


.145** 


221** 



Self-rated ability scales range from l=poor to 4=excellent 

* Correlation is significant at 0.05 level ** Correlation is significant at the 0.01 level 
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Mean math score 



While statistically significant, a number of the correlations in Table 9 are small, suggesting 
caution should be exercised in practical application of the findings. As an example of the 
positive correlation between self-rated abilities and test scores, GALE students who rated their 
ability to “solve mathematical equations” highly also scored better on mathematics items in 
particular and other subject areas as well. The following box plots further illustrate the 
relationship between two of the students’ self-rated abilities and test scores in math. 

Figure 1: Self-rated ability to understand the implications of scientific and technological 
compared to mean math score 




N 149 876 1025 302 



12 3 4 

l=Poor 2=Fair 3=Good 4=Excellent 



Figure 2: Self-rated ability to solve mathematical equations compared to mean math score 
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Similarly, higher test scores correlated with higher self-ratings of various abilities for all of the 

' / 

different subject areas. In particular, students’ self-rated ability to use logical reasoning to solve 
problems appeared to be evenly correlated with higher test scores across the five areas, 

I ' ' ' 

suggesting that the ability to use reasoning to solve problems is useful across a broad range of 
fields. On the other hand, a negative correlation was observed between students’ self-rated 
ability to understand art, classical music, and drama and their performance on the mathematics 
section. This finding may speak to the differing nature of humanities and more applied fields 
such as mathematics. In general, students appeared to have a sense of their relative knowledge 
and abilities and that sense was reflected in test scores for each of these areas 

The General Academic Learning Experience (GALE) study demonstrates a procedure by 
which cohort-level liberal arts student knowledge can be assessed as a function of student 
characteristics such as the number of courses completed, reasons for attending college, and self- 
rated abilities of students. While there was not a perfect correlation between student 
characteristics and subject area performance, students who answered at least seven of ten 
questions correctly in a particular subject area tended to differ from those students who answered 
three or fewer questions correctly. 



High and low scoring student profiles 

For example, students who answered seven or more English questions correctly (n=1102) 
had taken an average of three more college courses and one more English class than their lower 
scoring peers (n=533). Students answering seven or more questions correctly in humanities 
(n=329) were older (27 versus 25 years of age) and had taken three more college courses 
including one more humanities course than their lower scoring peers (n=1286). Students 
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answering seven or more math questions correctly (n=980) were younger than lower scoring 
students (n=753) with a median age of 20 years old compared to 22, had taken a greater number 
of college courses (a median of 12 versus 7 for low-scoring peers) including one more math class 
than, their lower scoring peers. Also, high scoring math students were more likely to name 
preparing for transfer to a four-year college or university as the most important reason for 
attending college at this time (71 percent compared to 53 percent of lower scoring students).. 
Students who had answered seven or more of ten science questions correctly (n=960) had taken 
one more science course on average than students answering three or fewer questions correctly 
(n=650). High scoring science students (n=540) most often had taken one more science course 
and rated their ability to understand different political ideologies and their ability to understand 
implications of scientific/technological developments higher than lower scoring students 
(n=936). Students who answered seven or more social science items correctly (n=650) followed 
similar though somewhat weaker patterns of differentiation seen in the subject areas above 
compared to lower scoring students (n=960). Students who scored higher in social science 
typically had taken more courses overall, more courses in social science, and more often rated 
themselves as better at a skill (in this case, “to understand different political ideologies”) 
compared to lower scoring students. 

Humanities subject area 

Student scores on GALE humanities items appear unresponsive to an increased number 
of general courses taken. On the other hand, mean GALE humanities scores did increase as 
students reported taking more humanities courses (see Table 6) which suggests the test may be a 
good measure of humanities knowledge. However, as student scores in the humanities are 
consistently the lowest of any GALE subject area, it seems likely that the humanities subject area 
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questions are either too difficult, too specific, or are not linked as firmly to the content of courses 
that students are talcing as are the other subject area questions. 

^ ■ I 

Humanities courses cover a broad range of content (e.'g. foreign languages, history, 
literature, art history/appreciation, music history/appreciation, cultural anthropology, religious 
studies, cultural geography and philosophy). Knowledge taken from one humanities course may 
not generalize to another as easily as in math or other more focused subject areas (see Table 1). 
While interdisciplinary humanities courses might help students respond correctly to a wider 
range of introductory level humanities questions, interdisciplinary courses are relatively rare. Of 
the 25 percent of total student enrollment for all humanities courses in 1998, less than one 
percent enrolled in interdisciplinary humanities courses (Cohen, 2002, p. 322). Thus, it seems 
reasonable that a student who has taken one or two distinct humanities courses may not score 
much higher on a broad-based test than students who have taken no courses. 

In addition, the predictive power of the GALE humanities section was structurally 
weakened by selection of topics and content. Specifically, students were asked to include 
foreign language courses when reporting the number of humanities classes they had taken, but no 
foreign language items were included on GALE. In future implementations, students should not 
be asked to include foreign language courses in the humanities count unless the humanities 
subject area includes foreign language items (see Appendix C for further discussion about error 
in construction of the humanities section on one of the five forms of the instrument). 

Limitations and Future Directions 

The GALE instrument is a reliable and a valid tool that differentiates in statistically 
significant ways among students who showed varying degrees of knowledge in the liberal arts. 
However, the practical significance of GALE may be questioned in several ways. For example. 



while students’ scores on the subject area tests show a positive statistically significant correlation 
with the total number of classes completed, the difference in mean scores between students who 
have taken zero courses and students who have taken ten or more courses is small, less than one 
point for any given subject area (Table 7). Further, there was a high rate of omitted answers 
(mean = 4 omitted answers) on the instrument, which dilutes the predictive power of tHe 
instrument. It is difficult to differentiate among students who omitted questions when we do, not 
know the reason for the omissions (e.g. time constraints, lack of motivation to try to do well on 
the test, lack of knowledge on that item, frustration, etc.). 

The lack of great variability in mean test scores as a function of the number of courses 
taken may be linked to factors at the level of the student, the faculty and the institution. For 
example, a student who has taken several remedial English and mathematics courses may very 
well score at a lower level than a student who does not need remedial work and has taken no 
courses at all. In the future, a means to differentiate between remedial and college level courses 
would be useful. In addition, the length of time that has elapsed since a student has completed 
related coursework may account for some of the low variability in test scores at the student level. 
In future iterations of GALE, it may help define student characteristics by asking students for 
numbers of courses they have taken within certain timeframes (e.g. two years or less; more than 
two years). 

- Implementation of this type of assessment also.relies heavily on faculty engagement and 

their ability to encourage full student participation. Enthusiastic faculty are likely to encourage 
students to do their best on the instrument. Methods of measuring and engaging faculty interest 
in tests such as GALE should be examined. 
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Finally, the number and type of courses offered by an institution should be assessed since 
it will affect the number and type of courses taken by students. For example, Cohen (2002) 

' ' ' ^ ■ I 

reports a trend m community college curriculum toward specialized courses and away from 
general survey courses. Perhaps students taking more specialized course knowledge in a subject 
area like humanities will not do as well on an instrument like GALE, which is designed to assess 
introductory and general knowledge in the liberal arts. 

In conclusion, we have presented a reliable and valid method to assess student learning in 
the liberal arts. A multiple-choice test like GALE also offers ease in assessing community 
college student knowledge. The convenience and speed of this method of assessment suits the 
transient nature and busy schedules of both community college students and faculty. The results 
of GALE might be used to help institutions assess their liberal arts curriculum and guide students 
toward behavior patterns and attitudes correlated with higher scores on this test. 

Of course there are also certain disadvantages accompanying use of content tests such as 
GALE. What is measured in a content test may be limited to relatively superficial knowledge or 
learning and there may be difficulty in interpreting the meaning of results due to student 
guessing and omission on items. Approaches such as interviews, portfolios, or writing samples 
could be used in conjunction with GALE to provide added depth and validity of assessment 

„ .CCohen,_2002;-Prus_& Johnson,_l&94)._In.truth,_while-no-one-method oLassessing-something as 

complex as stud ent learning is adequate or without disadvantages, G demonstrates one 
practical method that colleges may find useful in measuring student knowledge in the liberal arts. 
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APPENDIX A: Student Demographics Survey Items 
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General Academic Learning Exercise 




INFORMATION SHEET 

Dear Student: 

This survey is designed to gain information helpful in planning community college courses and programs. The information it will 
provide is confidential; your identity will not be revealed in any way. The information will not become part of your college record. 

Please use a idark pencil or pen to respond to each question. Make no stray marks on the form. Thank you for your participation. 

Dr. Arthur Cohen ; 

Director, Center for the Study of Community Colleges 

4. How many courses have you completed at 

this or any college? Do not include courses now ^ 

in progress. 

5. What is the highest degree you have already earned? 

O None O Bachelor 

O Associate O Master's or Doctorate 



1. Please fill in the oval corresponding to the letter printed 
on your question booklet. 

OA OB OC OD OE 

2. How old are you? 



3. Please mark your high school grade point average: 
(Mark only one) 

O A (4.0-3.75) O C (2.24-1 .75) 

O A-, B+ (3.74-3.25) O C- and below (1 .75-0) 

O B (3.24-2.75) O Not applicable 

O B-. C+ (2.74-2.25) 



6. What is the most important reason that you are attending 
college at this time? (Mark only one answer) 

O To prepare for transfer to a four-year college or university 
O To gain skills necessary to enter a specific occupation 
O To gain skills necessary to advance in a current occupation 
O To improve my English, reading or math skills 
O To satisfy a personal interest 



7. Indicate the number of college courses you have completed at this or any other college in each of the areas listed 
below. Do not include credit for courses now in progress. If none, indicate zero. 





0 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10+ 


Communications/English Composition 


O 


o 


o 


O 


o 


o 


o 


o 


o 


o 


o 


Humanities (e.g.. Art, Foreign Languages. Literature, 
Music and Philosophy) 


O 


o 


o 


O 


o 


o 


o 


o 


o 


o 


o 


Sciences (e.g.. Astronomy, Biology, Chemistry, 
Geology, Physics) 


O 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


Mathematics and Computer Science 


O 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


Social Sciences (e.g.. Anthropology. Economics, History, 
Psychology, Political Science, Sociology) 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 



8. Compared to other students at this college, how would you rate your ability to do the following? 





Poor 


Fair Good 


Excellent 




Understand the implications of scientific and 
technological developments 


o 


o 


o 


o 




Understand art, classical music, drama 


o 


o 


o 


o 




Understand different political ideologies 


o 


o 


o 


o 




Edit written material 


o 


o 


o 


o 




Solve mathematical equations 


o 


o 


o 


o 


FORM C 


^ ‘‘0 logical reasoning to solve problems 

IC 

“ BEST COPY AVAILABLE 


o 


o 
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APPENDIX B: Listing of subdisciplines and sample questions 

ENGLISH USAGE : reading,, writing, decoding and integrating communication through 

language 

SAMPLE QUESTION: 

I ' ' * 

Instructions: Two underlined sentences are followed by a question or a statement. Read the 
sentences, and then choose the best answer to the question or the best completion of the 
statement. 

The American prison system functions primarily to exact retribution. In Japan, the courts are less 
concerned with sending people to jail than they are with rehabilitating them. 

What does the second sentence do? 

A. It supports an idea found in the first sentence. 

B. It analyzes an idea stated in the first sentence. 

C. It states a contrast to the statement in the first sentence. 

D. It exemplifies an idea found in the first sentence. 

HUMANITIES : foreign languages, history, literature, art history/appreciation, music 

history/appreciation, cultural anthropology, religious studies, cultural 
geography and philosophy 

SAMPLE QUESTION: 

An interest in beauty and an inner sense of well-being in the presence of the beautiful is studied 
in that part of philosophy called: 

A. logic 

B. ethics 

C. epistemology 

D. aesthetics 
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MATHEMATICS : introductory and intermediate, applied (technology related), stats and 

probability, computer science and technology, remedial math 

SAMPLE QUESTION 

Simplify: {2a^bf 

A. 2a^b^ 

B. 4a b" 

C. 4flV 

D. 



SCIENCES : chemistry, biology, physics, earth and space sciences, engineering, 

environmental sciences, agriculture and natural resources, physical 
anthropology 

SAMPLE QUESTION 

Which of the following statements about enzymes is UNTRUE? 

A. They are proteins. 

B. Substrates attach to their active sites. 

C. They change the products that are produced during a reaction. 

D. They are reusable. 



SOCIAL SCIENCE : economics, archaeology, anthropology, urban planning, psychology, 

sociology, political science 

SAMPLE QUESTION 

Economics is primarily the study of: 

A. how to make money in the stock market 

B. how to operate a business successfully 

C. how society manages its scarce resources 

D. how the government spends money 
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APPENDIX C; Discriminating Power of Sub ject Area Test Sections 



Subject Area 


Classes completed in 
.subject area 


N ■ 


, Mean 
Score 


Standard 

Deviation 


English 


0 


793 


' 5.3 


.28 




1 or more 


.1482 


5.9 


.27 


Humanities 


, 0 


845 


3.5 


.26 




1 or more 


1388 


3.9 


.29 


Mathematics 


0 


778 


4.6 


.32 




1 or more 


1475 


5.6 


.34 


Science 


0 


751 


3.8 


.27 




1 or more 


1516 


4.4 


.29 


Social Science 


0 


751 


4.7 


.28 




1 or more 


1516 


5.4 


.31 



Subject area test items should separate students who know more about the subject area from 
student who know less. Using the number of classes completed as a proxy for student 
knowledge, it appears that four of the five subject areas in the table above are reliable. 

For example, the mean science scores obtained by students who have taken one or more science 
classes (5.9 ± .27) is greater than one standard deviation from those students who have taken 
zero science classes (5.3 ± .28). Three of the four remaining subject areas show this same clear 
pattern. The one exception is humanities. 

In addition to the discussion of why the humanities section may be unresponsive to students’ 
general course-taking patterns provided in the text, there was also an error in test construction in 
the humanities section of Form C. A series of linked questions offered five potential answers 
where the'answer sheet allowed for only four selections. This error invalidated 5 of the 10 
humanities items in this test form, leaving a total of 45 questions for this test form compared to 

50 items-on-the-Other_four_forms._Since-the.percent-of-humanities-questions.that_students 

answered correctly in the aggregate fell within the range of the percent of questions in other 
subject areas, we used the “percentage of items correct” rather than the “number of items 
answered correct” as the outcome variable. This adjustment increases the potential error for the 
humanities section of Form C since it effectively double-weights the remaining five items. A 
smaller available test base may also limit the stratification in scores observed among students 
who know more or less in humanities. 
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